Surgical guidance intersection display

ABSTRACT

A system and method for providing image guidance for placement of one or more medical devices at a target location. The system can determine one or more intersections between a medical device and an image region based at least in part on first emplacement data and second emplacement data. Using the determined intersections, the system can cause one or more displays to display perspective views of image guidance cues, including an intersection indicator in a virtual 3D space.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority benefit to U.S. ProvisionalApplication No. 62/091,238, which is hereby incorporated herein byreference in its entirety.

TECHNICAL FIELD

The systems and methods disclosed herein relate generally to computersystems facilitating medical device guidance through tissue by a medicalpractitioner.

BACKGROUND

Various medical device systems are available to aid a healthcareprovider to guide a medical device in a patient. The medical devicesystems can provide various image guidance cues to aid the healthcareprovider, and can also provide views of images of an imaged region andof virtual medical devices corresponding to physical medical devices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an embodiment of a system for image-guidedmedical procedures.

FIG. 2 is a diagram of an embodiment of a rendering of image guidancecues and medical display objects on a display.

FIGS. 3A, 3B, 3C are diagrams of an embodiment illustrating variousperspective views of a scene including at least one variance volume.

FIG. 4 is a flow diagram illustrative of an embodiment of a routineimplemented by the system to display the variance volume.

FIGS. 5A, 5B, and 5C are diagrams of an embodiment illustrating variousperspective views of a scene including an elliptical intersectionindicator.

FIGS. 6A, 6B, and 6C are diagrams of an embodiment illustrating variousperspective views of a scene including an obround-shaped intersectionindicator.

FIGS. 7A, 7B, and 7C are diagrams of an embodiment illustrating variousperspective views of a scene including multiple intersection indicators.

FIGS. 8A and 8B are diagrams of embodiments illustrating views ofdifferent scenes including an intersection indicator.

FIGS. 9A, 9B, and 9C are diagrams of an embodiment illustrating variousperspective views of a scene including multiple intersection indicatorsassociated with different virtual medical devices.

FIGS. 10A and 10B are diagrams of an embodiment illustrating variousperspective views of a scene including multiple display objects.

FIGS. 11A and 11B are diagrams illustrating an embodiment in which anintersection indicator is located outside an image region

FIG. 12 is a flow diagram illustrative of an embodiment of a routineimplemented by the system to display an intersection indicator.

FIG. 13 is a flow diagram illustrative of an embodiment of a routineimplemented by the system to display an intersection indicator.

DETAILED DESCRIPTION

Implementations disclosed herein provide systems, methods, and apparatusfor generating images facilitating medical device insertion into tissueby an operator. Certain embodiments pertain to a free-hand medicaldevice guidance system. The system can provide the healthcare providermanual control over the medical device, while making the spatialrelationships between the target, medical device and U/S image moreintuitive via a visual display. Using this visual feedback, the operatorcan adjust the medical device's position, orientation, or trajectory.Certain of the contemplated embodiments can be used in conjunction withsystems described in greater detail in U.S. patent application Ser. No.13/014,587, filed Jan. 26, 2011, entitled SYSTEMS, METHODS, APPARATUSES,AND COMPUTER-READABLE MEDIA FOR IMAGE MANAGEMENT IN IMAGE-GUIDED MEDICALPROCEDURES; U.S. patent application Ser. No. 13/753,274, filed Jan. 29,2013, entitled MULTIPLE MEDICAL DEVICE GUIDANCE (the '274 Application);U.S. patent application Ser. No. 14/212,933, filed Mar. 14, 2014,entitled MEDICAL DEVICE GUIDANCE; and U.S. patent application Ser. No.14/872,930, entitled AFFECTED REGION DISPLAY, filed Oct. 1, 2015; eachof which is hereby incorporated herein by reference in its entirety.

The system can aid the healthcare provider in guiding one or moremedical devices through the tissue of the patient and/or placing themedical devices, and can be used for treatment of tumors, fibroids orcysts, with bipolar radiofrequency medical device ablation, multiplemicrowave medical devices, electroporation, and/or electrochemotherapysystems. It can also be used for nerve or muscle stimulation or sensing(electrodes in the spine, brain). The system can be used during opensurgery, laparoscopic surgery, endoscopic procedures, biopsies, and/orinterventional radiology procedures.

The system can be used in conjunction with live intraoperativeultrasound (U/S), pre-operative CT, or any cross-sectional medicalimaging modality (e.g. MRI, OCT, etc.). In addition, the system can usea variety of techniques to determine the position and/or orientation ofone or more medical devices. For example, the system can use the NDIAurora magnetic system, the Ascension MedSafe system, NDI Polarisoptical system, etc. In some embodiments, a position sensor can beembedded inside, or affixed to each medical device, at the tip, alongthe shaft, and/or on the handle. Sensors can be built into the medicaldevices or attached after manufacturing, as described in greater detailin U.S. application Ser. No. 14/212,184, filed Mar. 14, 2014, entitledSENSOR MOUNT, incorporated herein in its entirety.

Each medical device can be associated with one or more sensors, whichcan continually, or repeatedly, report position and/or orientation, or asingle sensor can be used for all the medical devices. In embodimentswhere one sensor is used, the healthcare provider can attach the sensorto the particular medical device that she is intentionallyrepositioning, and then, once she has placed that medical device, shecan remove the sensor and attach it to the next medical device she isrepositioning. In some embodiments, the medical devices can bemanipulated by the healthcare provider. In certain embodiments, thesystem can be used with a robotic manipulator, where the robot controlsthe medical devices.

In some embodiments, the handles of medical devices can have push-buttonswitches, to allow the user to select a medical device, indicate atissue target, etc. The handle can also have an indicator light toindicate to the users which medical device is selected. Finally, thehandle can have an encoder to detect how much length of electrode hasbeen exposed by the user, and report this information to the guidancesystem and therapeutic generator

Image Guidance Systems

FIG. 1 is a diagram illustrating an embodiment of an environment 100 forimage management in image-guided medical procedures. In the illustratedembodiment, the environment 100 includes a display 102 displaying animage 150, an image guidance unit 104, a position sensing unit 106, asurgical system 108, imager 110, surgical instruments 112, 155, apatient 116, a stand 118, and a table 120. In some embodiments, theimage guidance system 101 can include any one or any combination of thedisplay 102, the image guidance unit 104, the position sensing unit 106,the surgical system 108, the imager 110, the surgical instruments 112,155, the stand 118, and/or the table 120.

In some embodiments, the position sensing unit 106 can track surgicalinstruments 112, 114, also referred to herein as medical devices 112,114, within a tracking area and provide data to the image guidance unit104. The medical devices 112, 114 can include invasive medical devices,such as, but not limited to, biopsy needles, ablation needles, surgicalneedles, nerve-block needles, or other needles, electrocautery device,catheters, stents, laparoscopes or laparoscopic cameras, ultrasoundtransducers, or other instruments that enter a part of the body, andnon-invasive medical devices that do not enter the body, such as, butnot limited to, ultrasound transducers, probes, or other externalimaging devices, etc. The medical devices 112, 114 can also includemedical imaging devices that provide or aid in the selection of medicalimages for display. In some embodiments, the medical imaging device canbe any device that is used to select a particular medical image fordisplay. The medical imaging devices can include invasive medicaldevices, such as laparoscopic cameras, and non-invasive medical devices,such as external ultrasound transducers.

Although only two surgical instruments 112, 114 are shown in FIG. 1, itwill be understood that additional surgical instruments can be trackedand associated data can be provided to the image guidance unit 104. Theimage guidance unit 104 can process or combine the data and show imageguidance data on display 102. This image guidance data can be used by ahealthcare provider to guide a procedure and improve care. There arenumerous other possible embodiments of system 101. For example, many ofthe depicted components can be joined together to form a singlecomponent and can be implemented in a single computer or machine.Further, additional position sensing units can be used in conjunctionwith position sensing unit 106 to track relevant surgical instruments112, 114, as discussed in more detail below. Additional imagers 110 canbe included, and combined imaging data from the multiple imagers 110 canbe processed by image guidance unit 104 and shown on display 102.Additionally, two or more surgical systems 108 can be used.

Information about and from multiple surgical systems 108 and attachedsurgical instruments 112 (and additional surgical instruments not shown)can be processed by image guidance unit 104 and shown on display 102.These and other possible embodiments are discussed in more detail below.It will be understood that any combination of the display objects, imageguidance cues, etc., described herein can be displayed concurrently, orsimultaneously. Further, reference to displaying objects “concurrently”and/or “simultaneously” is to be interpreted broadly and may refer todisplaying objects in such a way that to a human observer the objectsare visible at the same time.

Imager 110 can be communicatively coupled to image guidance unit 104. Insome embodiments, imager 110 can be coupled to a second display unit(not shown). The second display unit can display imaging data fromimager 110. The imaging data displayed on display 102 and displayed onsecond display unit can be the same or different. In some embodiments,the imager 110 can be an ultrasound machine 110, the medical device 114can be a movable imaging unit, such as an ultrasound transducer 114 orultrasound probe 114, and the second display unit can be a displayassociated with the ultrasound machine 110 that displays the ultrasoundimages from the ultrasound machine 110. In some embodiments, a movableimaging unit 114 can be communicatively coupled to image guidance unit104. The movable imaging unit 114 can be useful for allowing a user toindicate what portions of a first set of imaging data are to bedisplayed. For example, the movable imaging unit 114 can be anultrasound transducer 114, a needle or other medical device, forexample, and can be used by a user to indicate what portions of imagingdata, such as a pre-operative CT scan, to show on a display 102 as image150. Further, in some embodiments, there can be a third set ofpre-operative imaging data that can be displayed with the first set ofimaging data.

In some embodiments, a navigation system comprises a display 102 and aposition sensing unit 106 communicatively coupled to image guidance unit104. In some embodiments, position sensing unit 106, display 102, andimage guidance unit 104 are coupled to the stand 118. Image guidanceunit 104 can be used to produce images 150 that are displayed on display102. The images 150 produced on display 102 by the image guidance unit104 can be determined based on ultrasound or other visual images fromthe first surgical instrument 112 and second surgical instrument 114.

In the illustrated embodiment, the image 150 includes a 2D viewing area152 and a 3D viewing area 154 (which can also be referred to as avirtual 3D space) each of which includes various display objects. In the2D viewing area, some or all of the display objects can be displayed as2D objects. However, it will be understood that some of the displayobjects in the 2D viewing area can be displayed as 3D objects. In the 3Dviewing area 154, some or all of the display objects are displayed as 3Dobjects. Furthermore, the display objects in the 3D viewing area can bedisplayed in a perspective based at least in part on a point-of-viewlocation. In the illustrated embodiment, the display objects include, animage region 156 with an ultrasound image 157, a virtual medical device158 corresponding to the first surgical instrument 112, a virtualimaging device 160 corresponding to the second surgical instrument 114,intersection indicator 162, trajectory indicator 164, variance volumeindicator 166, and shaded region 168. It will be understood that anycombination of the aforementioned display objects can be displayed inthe 2D viewing area and/or 3D viewing area as desired.

As a non-limiting example, if the first surgical instrument 112 is anablation needle 112 and the second surgical instrument 114 is anultrasound probe 114, then images 150 produced on display 102 caninclude the images, or video, from the ultrasound probe 114 (e.g., imageslice 156) combined with other medical display objects and imageguidance cues, such as projected medical device drive (e.g., trajectoryindicators 164) or projected ablation volume (not shown), determinedbased on the emplacement of ablation needle 112. If the first surgicalinstrument 112 is an ultrasound probe 112 and the second surgicalinstrument 114 is a laparoscopic camera 114, then images 150 produced ondisplay 102 can include the video from the laparoscopic camera 114combined with ultrasound data superimposed on the laparoscopic image.More surgical instruments can be added to the system. For example, thesystem can include an ultrasound probe, ablation needle, laparoscopiccamera, stapler, cauterizer, scalpel and/or any other surgicalinstrument or medical device. The system can also process and/or displaycollected data, such as preoperative CT scans, X-Rays, MRIs, laserscanned 3D surfaces etc.

The term “emplacement” as used herein is a broad term and may refer to,without limitation, position and/or orientation or any other appropriatelocation information. The term “pose” as used herein is a broad termencompassing its plain and ordinary meaning and may refer to, withoutlimitation, position and orientation or any other appropriate locationinformation. In some embodiments, the imaging data obtained from one orboth of surgical instruments 112 and 114 can include other modalitiessuch as a CT scan, MRI, open-magnet MRI, optical coherence tomography(“OCT”), positron emission tomography (“PET”) scans, fluoroscopy,ultrasound, or other preoperative, or intraoperative 2D or 3D anatomicalimaging data. In some embodiments, surgical instruments 112 and 114 canalso be scalpels, implantable hardware, or any other device used insurgery. Any appropriate surgical system 108 or imager 110 can becommunicatively coupled to the corresponding medical instruments 112 and114.

As noted above, the images 150 produced can also be generated based onlive, intraoperative, or real-time data obtained using the secondsurgical instrument 114, which is communicatively coupled to imager 110.The term “real time” as used herein is a broad term and has its ordinaryand customary meaning, including without limitation instantaneously ornearly instantaneously. The use of the term real time can also mean thatactions are performed or data is obtained with the intention to be usedimmediately, upon the next cycle of a system or control loop, or anyother appropriate meaning. Additionally, as used herein, real-time datacan be data that is obtained at a frequency that would allow ahealthcare provider to meaningfully interact with the data duringsurgery. For example, in some embodiments, real-time data can be amedical image of a patient that is updated one time per second. In someembodiments, real-time data can be ultrasound data that is updatedmultiple times per second.

The surgical instruments 112, 114 can be communicatively coupled to theposition sensing unit 106 (e.g., sensors embedded or coupled to thesurgical instruments 112, 114 can be communicatively coupled with theposition sensing unit 106). The position sensing unit 106 can be part ofimager 110 or it can be separate. The position sensing unit 106 can beused to determine the emplacement of first surgical instrument 112and/or the second surgical instrument 114. In some embodiments, theposition sensing unit 106 can include a magnetic tracker and/or one ormore magnetic coils can be coupled to surgical instruments 112 and/or114. In some embodiments, the position sensing unit 106 can include anoptical tracker and/or one or more visually-detectable fiducials can becoupled to surgical instruments 112 and/or 114. In some embodiments, theposition sensing unit 106 can be located below the patient. In suchembodiments, the position sensing unit 106 can be located on or belowthe table 120. For example, in embodiments where the position sensingunit 106 is a magnetic tracker, it can be mounted below the surgicaltable 120. Such an arrangement can be useful when the tracking volume ofthe position sensing unit 106 is dependent on the location of theposition sensing unit 106, as with many magnetic trackers. In someembodiments, magnetic tracking coils can be mounted in or on the medicaldevices 112 and 114.

In some embodiments, the position sensing unit can determine one or morex, y, z coordinates and/or the quaternions (e.g., yaw, pitch, and/orroll) of device trackers associated with one or more of the medicaldevices 112, 114. In some embodiments, the position sensing unit 106 canbe an electromagnetic measurement system (e.g., NDI Aurora system) usingsensor coils for device trackers attached to the first and/or secondsurgical devices 112, 114. In some embodiments, the position sensingunit 106 can be an optical 3D tracking system using fiducials. Suchoptical 3D tracking systems can include the NDI Polaris Spectra, Vicra,Certus, PhaseSpace IMPULSE, Vicon MX, InterSense IS-900, NaturalPointOptiTrack, Polhemus FastTrak, IsoTrak, or Claron MicronTracker2. In someembodiments, the position sensing unit 106 can each be an inertial 3Dtracking system comprising a compass, accelerometer, tilt sensor, and/orgyro, such as the InterSense InertiaCube or the Nintendo Wii controller.In some embodiments, the position sensing unit 106 can be attached to oraffixed on the corresponding surgical device 112 and 114.

In some embodiments, the position sensing units 106, can include sensingdevices such as the HiBall tracking system, a GPS device, or signalemitting device that would allow for tracking of the position and/ororientation (e.g., emplacement) of the device tracker (also referred toas an emplacement sensor). In some embodiments, a position sensing unit106 can be affixed to either or both of the surgical devices 112, 114.The surgical devices 112 or 114 can be tracked by the position sensingunit 106. A room coordinate system reference, such as the display 102can also be tracked by the position sensing unit 106 in order todetermine the emplacements of the surgical devices 112, 114 with respectto the room coordinate system. Devices 112, 114 can also include or havecoupled thereto one or more accelerometers, which can be used toestimate movement, position, and location of the devices.

In some embodiments, the position sensing unit 106 can be an AscensionFlock of Birds, Nest of Birds, driveBAY, medSAFE, trakSTAR, miniBIRD,MotionSTAR, pciBIRD, or Calypso 2D Localization System and devicetrackers attached to the first and/or second medical devices 112, 114can be magnetic tracking coils.

The term “device tracker” (also referred to as an emplacement sensor),as used herein, is a broad term encompassing its plain and ordinarymeaning and includes without limitation all types of magnetic coils orother magnetic field sensing devices for use with magnetic trackers,fiducials or other optically detectable markers for use with opticaltrackers, such as those discussed above and below. In some embodiments,the device trackers can be implemented using optical position sensingdevices, such as the HiBall tracking system and the position sensingunit 106 can form part of the HiBall tracking system. Device trackerscan also include a GPS device or signal emitting device that allows fortracking of the position and/or orientation of the device tracker. Insome embodiments, a signal emitting device might include aradio-frequency identifier (RFID). In such embodiments, the positionsensing unit 106 can use the GPS coordinates of the device trackers orcan, for example, triangulate the radio frequency signal being emittedby the RFID associated with device trackers. The tracking systems canalso include one or more 3D mice.

Furthermore, the system 101 can use the emplacement data associated withthe device trackers (e.g., received from the device trackers or from theposition sensing unit 106) to determine other emplacement information,including, but not limited to the emplacement of a trajectory, an imageplane, image region, imaged region, and/or one or more intersections,etc. The determined emplacement information can be used to generate anddisplay the various image guidance cues, such as, but not limited to,the intersection indicator 162, the trajectory indicator 164, thevariance volume 166, the shaded region 168, the image region 156 withthe image 157, etc.

In some embodiments, the imaged region can correspond to the tissue orregion that is imaged (area and/or volume) by the medical device 114. Insome cases, the image plane can correspond to the plane at which themedical device 114 acquires an image and/or a plane in the virtual 3Dspace that is associated therewith. In certain cases, the image regioncan correspond to the region (area and/or volume) at which the medicaldevice 114 acquires an image and/or to a region in the virtual 3D spaceassociated therewith. For example, in some cases, image data acquired bythe medical device 114 in the medical device's image region can bemapped to a corresponding virtual image region in the virtual 3D space.The image region may also be referred to as an image slice and/or imageslab. Furthermore, in some embodiments, the image region can include atleast a portion of the image plane.

The emplacement of the image plane, image region (area and/or volume),and/or imaged region can also be determined based at least in part onthe operating parameters of the medical device 114. For example, theoperating parameters can indicate what portion of the medical device 114will capture an image (e.g., emit ultrasonic waves), as well as thedimensions of the image region and/or imaged region (e.g., height,width, and/or depth of the image that will be acquired), as well as theimage region.

Images 150 can be produced based on intraoperative or real-time dataobtained using first surgical instrument 112, which is coupled to firstsurgical system 108. In the illustrated embodiment of FIG. 1, the firstsurgical system 108 is shown as coupled to image guidance unit 104. Thecoupling between the first surgical system 108 and image guidance unit104 may not be present in all embodiments. In some embodiments, thecoupling between first surgical system 108 and image guidance unit 104can be included where information about first surgical instrument 112available to first surgical system 108 is useful for the processingperformed by image guidance unit 104. For example, in some embodiments,the first surgical instrument 112 can be an ablation needle 112 andfirst surgical system 108 can be an ablation system 108. In someembodiments, it can be useful to send a signal about the relativestrength of planned ablation from ablation system 108 to image guidanceunit 104 so that the image guidance unit 104 can show a predictedablation volume. In other embodiments, the first surgical system 108 isnot coupled to image guidance unit 104. Example embodiments includingimages and graphics that can be displayed are included below.

In some embodiments, the display 102 displays 3D images to a user, suchas a healthcare provider. Stereoscopic 3D displays separate the imageryshown to each of the user's eyes. This can be accomplished by astereoscopic display, a lenticular auto-stereoscopic display, or anyother appropriate type of display. The display 102 can be an alternatingrow or alternating column display. Example alternating row displaysinclude the Miracube G240S, as well as Zalman Trimon Monitors.Alternating column displays include devices manufactured by Sharp, aswell as many “auto-stereoscopic” displays (e.g., Philips). In someembodiments, Sony Panasonic 3D passive displays and LG, Samsung, and/orVizio 3D TVs can be used as well. Display 102 can also be a cathode raytube. Cathode Ray Tube (CRT) based devices, can use temporal sequencing,showing imagery for the left and right eye in temporal sequentialalternation. This method can also be used projection-based devices, aswell as by liquid crystal display (LCD) devices, light emitting diode(LED) devices, and/or organic LED (OLED) devices.

In certain embodiments, the display 102 can be a head mounted display(HMD) worn by the user in order to receive 3D images from the imageguidance unit 104. In such embodiments, a separate display, such as thepictured display 102, can be omitted. The 3D graphics can be producedusing underlying data models, stored in the image guidance unit 104 andprojected onto one or more 2D planes in order to create left and righteye images for a head mount, lenticular, or other 3D display. Theunderlying 3D model can be updated based on the relative emplacements ofthe various devices 112 and 114, as determined by the position sensingunit(s) 106, and/or based on new data associated with the devices 112and 114. For example, if the second medical device 114 is an ultrasoundprobe, then the underlying data model can be updated to reflect the mostrecent ultrasound image. If the first medical device 112 is an ablationneedle, then the underlying model can be updated to reflect any changesrelated to the needle, such as power or duration information. Anyappropriate 3D graphics processing can be used for rendering includingprocessing based on OpenGL, Direct3D, Java 3D, etc. Whole, partial, ormodified 3D graphics packages can also be used, such packages including3DS Max, SolidWorks, Maya, Form Z, Cybermotion 3D, VTK, Slicer, or anyothers. In some embodiments, various parts of the needed rendering canoccur on traditional or specialized graphics hardware. The rendering canalso occur on the general CPU, on programmable hardware, on a separateprocessor, be distributed over multiple processors, over multiplededicated graphics cards, or using any other appropriate combination ofhardware or technique.

One or more components, units, devices, or elements of variousembodiments can be packaged and/or distributed as part of a kit. Forexample, in one embodiment, an ablation needle, one or more devicetrackers, 3D viewing glasses, and/or a portion of an ultrasound wand canform a kit. Other embodiments can have different elements orcombinations of elements grouped and/or packaged together. Kits can besold or distributed separately from or with the other portions of thesystem.

One will readily recognize that there are numerous other examples ofimage guidance systems which can use, incorporate, support, or providefor the techniques, methods, processes, and systems described herein.

Depicting Surgical Instruments

It can often be difficult to discern the content of a 3D scene from a 2Ddepiction of it, or even from a 3D depiction of it. Therefore, variousembodiments herein provide image guidance that can help the healthcareprovider better understand the scene, relative emplacements or poses ofobjects in the scene and thereby provide improved image guidance.

FIG. 2 illustrates a perspective view of a virtual rendering 202 of asurgical instrument 242 being displayed on a screen 220 with aperspective view of a medical image 204. In some embodiments, the screen220 can correspond to the screen of the display 102, which can beimplemented using a TV, computer screen, head-mounted display,projector, etc. In the illustrated embodiment, the rendered surgicalinstrument 202 displayed on the screen 220 corresponds to the ablationneedle 242. A wire 246 connecting the ablation needle 242 to an ablationsystem (not shown) is also depicted in FIG. 2.

Although only one virtual surgical instrument 202 is displayed, it willbe understood that multiple medical devices can be tracked and displayedconcurrently, or simultaneously, on screen 220, as described in greaterdetail in the '274 Application, previously incorporated by reference.For example, a virtual rendering of the medical imaging device 222 canbe displayed.

The virtual surgical instrument 202 can be displayed in a virtual 3Dspace with the screen 220 acting as a window into the virtual 3D space.Thus, as the surgical instrument 242 is moved to the right with respectto a point-of-view location (e.g., the location of the point-of-view forviewing the 3D space), the virtual surgical instrument 202 can also moveto the right. Similarly, if the surgical instrument 242 is rotated 90degrees so that the tip of the surgical instrument is pointing away fromthe point-of-view location (e.g., at the screen 220), the virtualsurgical instrument 201 will likewise show the change in orientation,and show the tip of the virtual surgical instrument 202 in thebackground and the other end of the virtual surgical instrument 202 inthe foreground. In some embodiments, as described in greater detail inU.S. application Ser. No. 14/212,933, incorporated herein by referencein its entirety, the point-of-view location can be a fixed location,such as a predetermined distance/angle from the screen 220 or stand 118and or a location configured by the user; or the point-of-view locationcan by dynamic. For example, the system can track a user in real-timeand determine the point-of-view location based at least in part on thetracked location of the user.

Some models of medical devices have markings such as bands around theshaft (to indicate distance along the shaft), and a colored region 203near the tip to indicate from where the radio frequency or microwaveenergy is emitted in the case of an ablation probe. Healthcare providersperforming medical device procedures are often familiar with thesemarkings and can use them to help understand the spatial relationshipbetween the medical device and anatomy. In some embodiments, the makeand model of the medical device 242 is known to the image guidancesystem and the virtual medical device 202 displayed in display 220 canresemble medical device 242. The features of medical devices that can berendered in the scene include the overall shape (diameter, crosssectional shape, curvature, etc.), color, distance markers, visuals orechogenic fiduciary markers, the state of deployable elements such astines, paddles, anchors, resection loops, stiffening or steerablesleeves, temperature, radiation, light or magnetic field sensors, lens,waveguides, fluid transfer channels, and the like.

The type of medical device being used can be input into the imageguidance system 101, can be a system default, can be detected by acamera or other device, can be received as data from an attached medicaldevice, such as surgical system 108 in FIG. 1, or the information can bereceived in any other appropriate manner. Displaying on display 220, avirtual surgical instrument that resembled the surgical instrument 242can help healthcare providers associate the image guidance data with thereal world and can provide more familiar guidance information to ahealthcare provider, thereby further aiding the healthcare provider inthe guidance task. For example, the healthcare provider can see thefamiliar markings on the medical device being displayed on the display220 and therefore be familiar with the distance and relative placementof the displayed medical device with respect to other data, such as atumor 212 seen in a rendered ultrasound image 204, 205. This knowledgeof relative placement of items being displayed can help the healthcareprovider move the medical device 242 into place.

Consider an embodiment in which the virtual surgical instrument 202 inthe display 220 is an ablation needle depicting the portion of theneedle that will perform the ablation, for example, the portion thatemits the radio or microwave energy. If the display 220 also includesultrasound data, then the doctor can be able to find the tumor 212 shewishes to ablate by moving the ultrasound probe around until she spotsthe tumor 212. In various embodiments, she will be able to see thedisplayed ultrasound data and its location relative to the displayedmedical device with the markings. She can then drive the medical deviceuntil she sees, on display 220, that the emitter-portion of the medicaldevice encompasses the tumor in the ultrasound, also seen on display220. When she activates the ablation, she can then be more certain thatshe has ablated the correct portion of the tissue. Various embodimentsof this are discussed below.

As another example, consider the physical markings that can be on theinstruments themselves. These markings can help orient a healthcareprovider during use of the instrument. In some embodiments, the imageguidance unit can represent these markings in the images displayed inthe display. For example, certain ultrasound transducers are built withan orientation mark (e.g., a small bump) on one side of the transducingarray. That mark can also be shown in the ultrasound image on thescanner's display, to help the healthcare provider understand where thescanned anatomical structures shown on screen are located under thetransducer, inside the patient. In some embodiments, the image guidancesystem can display a symbolic 3D representation of the orientation markboth next to the motion-tracked ultrasound slice (e.g., moving with thedisplayed ultrasound slice) and next to the 2D view of the ultrasoundslice also displayed by the system. An example of this is displayed inFIG. 2, where a small rectilinear volume 214 corresponding to a featureon an ultrasound probe is shown both in proximity to the ultrasoundslice displayed in the 3D view and the ultrasound slice displayed in a2D view.

It will be understood that an image slice or image slab can also referto image data received from an imaging device, such as an ultrasoundtransponder. In some embodiments, the image data can correspond to across-section of tissue having a certain thickness. In some instances,the imaging device can compact the image data, and/or treat the imagedata as 2D data, such that there is no perceived thickness. In certainembodiments, when the image slice is displayed in a 3D view, the systemcan treat the image slice as a 2D or quasi 2D object. In suchembodiments, the system can cause the image slice to have little to noperceptible thickness. Accordingly, in certain embodiments, when theimage slice is oriented orthogonally or perpendicularly with respect tothe point-of-view location, the system can cause the display to displaynothing or a line having a relatively small thickness, such as a fewpixels, etc. In some cases, the number of pixels used to display therelatively small thickness of the image slice can correspond to the sizeof the display. For example, more pixels can be used for a largerdisplay and fewer pixels can be used for a smaller display, etc.

Other embodiments can track and display other types of instruments andtheir features. For example, a healthcare provider may want to track oneor more of a scalpel, a biopsy, a cauterizer (including anelectrocauterizer and Bovies), forceps, cutting loops on hysteroscopes,harmonic sheers, lasers (including CO₂ lasers), etc. For example, invarious embodiments, the following devices can be tracked and variousaspects of their design displayed on display 220: Olympus™ OES ProHystero-Resectoscope, SonoSurg Ultrasonic Surgical System Olympus™ GF-UC160 Endoscope Wallus™ Embryo Transfer Catheter AngioDynamics®NanoKnife™, VenaCure™ laser, StarBurst, Uniblade, Habib® Resector Bovie™Electrodes, Covidien Evident™, Cool-tip™ Ablation Antennas, Opti4™Electrodes Microsulis MEA (microwave endometrial ablation), AcculisHalt™ Medical System Optimed BigLumen Aspiration Catheter OptimedOptipure Stent Central venous catheterization introducer medical device(such as those made by Bard and Arrow).

Once tracked, a healthcare provider is able to see image guidance dataon display 220 that will allow her to know the relative pose, location,or emplacement of the tracked instrument(s) with respect to one anotheror with respect to imaging data and will be able to see, on display 220,the features of the instrument rendered in the scene.

Depicting Medical Device Placement, Trajectory, and Other Image GuidanceCues

In certain procedures, the system can provide image predictioninformation related to the surgical instruments as image guidance cues.In the context of scalpel movement, this can be the location that thescalpel will hit if a healthcare provider continues to move the scalpelin a particular direction. In the context of ablation or biopsies, thiscan be the projected medical device placement if it is driven along itscentral axis, which is also referred to herein as a longitudinal axis.

FIG. 2 further illustrates an embodiment of a projected needle drive 208(also referred to as a trajectory indicator) as an image guidance cue.If a healthcare provider is driving an ablation needle 242 into tissue(not pictured), then she can know where the medical device will bedriven. In some embodiments, the projected drive 208 of a medical devicecan be depicted on the display 220 and can show the healthcare providerthe projected path 208 that the medical device 242 will take if it isdriven along its central axis. Although the trajectory of only onemedical device is displayed, it will be understood that the trajectoryof multiple medical devices can be determined and displayedsimultaneously on screen 220, as described in greater detail in the '274Application.

In some embodiments, to implement the trajectory indicators 208, theimage guidance system can draw a number of rings about the axis of themedical device shaft, extrapolated beyond its tip, as depicted in FIG.2. A healthcare provider can view and manipulate the emplacement of themedical device 242 and its expected drive projection (via its displayedprojected trajectory) before it enters the patient's tissue. In someembodiments, this is accomplished by the doctor positioning the virtualrings in the drive projection such that they are co-incident (or passthrough) the ultrasound representation of a target, such as a tumor thatthe doctor has spotted in the ultrasound. This can allow the healthcareprovider to verify that the medical device 242 is properly aimed at thetarget and can drive the medical device 242 forward into the tissue suchthat it reaches its desired target or destination. For example, if thedoctor identifies a tumor 212 in the ultrasound image, she can align theablation needle 242 such that the drive projection rings on display 220intersect or otherwise indicate that the medical device, if drivenstraight, will reach the tumor 212.

The rings can, in some embodiments, be spaced at regular (e.g., 0.5, 1,or 2 cm) intervals to provide the healthcare provider with visual orguidance cues regarding the distance from the medical device tip to thetargeted anatomy. In some embodiments, the spacing of the rings canindicate other aspects of the data, such as the drive speed of themedical device, the density of the tissue, the distance to a landmark,such as the ultrasound data, or any other appropriate guidance data orproperty. In some embodiments, the rings or other trajectory indicatorscan extend beyond the medical device tip, by a distance equal to thelength of the medical device-shaft. This way, the user knows if themedical device is long enough to reach the target--even before the tipenters the patient. That is, in some embodiments, if the rings do notreach the target with the tip still outside the body, then the tip willnot reach the target even when the entire length shaft is inserted intothe body.

Other display markers can be used to show trajectory, such as a dashed,dotted, or solid line, transparent medical device shaft, point cloud,wire frame, etc. In some embodiments, three-dimensional rings can beused and provide depth cues and obscure little of the ultrasound image.Virtual rings or other virtual markers can be displayedsemi-transparently, so that they obscure less of the ultrasound imagethan an opaque marker would.

Other prediction information can also be displayed as image guidancecues. For example, if a scalpel is being tracked by the image guidancesystem, then a cutting plane corresponding to the scalpel can bedisplayed (not pictured). Such a cutting plane can be coplanar with theblade of the scalpel and can project from the blade of the scalpel. Forexample, the projected cutting plane can show where the scalpel wouldcut if the doctor were to advance the scalpel. Similar predictioninformation can be estimable or determinable for cauterizers, lasers,and numerous other surgical instruments.

Furthermore, the data from two or more devices can be combined anddisplayed based on their relative emplacements or poses. For example,the rendered ultrasound image 204 can be displayed on the image plane(e.g., in the image region) with respect to the virtual medical device202 on the display 220 in a manner that estimates the relativeemplacements or poses of the medical imaging device 222 and the medicaldevice 242. As illustrated in FIG. 2, the image guidance cues associatedwith the virtual medical device 202, including the affected regionindicator 206 and trajectory indicators 208, are shown spatially locatedwith the rendered ultrasound image 204 on display 220.

In addition, the display 220 can include another image guidance cue inthe form of an intersection indicator 210 that indicates where thevirtual ablation medical device 202 (and/or its axis and/or itstrajectory) intersects the ultrasound image 204. In some embodiments,the intersection indicator 210 can be displayed before the medicaldevice is inserted, thereby allowing the healthcare provider to seewhere the medical device will intersect the image, or imaged region. Aswill be described in greater detail below, in some cases, due touncertainties related to the emplacement of the medical devices, thesystem can use a variance parameter to determine and display theintersection indicator 210.

In the illustrated embodiment, a tumor 212 appears in the ultrasoundimage, or rendered ultrasound image 204, and the virtual ablation needle202 is shown driven through the tumor 212. As described in greaterdetail in U.S. application Ser. No. 14/872,930 (the '930 Application),incorporated herein by reference in its entirety, the displayed affectedregion (or affected region indicator) 206 can indicate what region orvolume would be affected when the medical device 242 is operated. In theillustrated embodiment, the displayed affected region 206 can estimatewhere ablation would occur if the tissue were ablated at that time. Ascan be seen, in the illustrated embodiment, the displayed affectedregion 206 appears to cover the tumor displayed in the ultrasound image.

It will be understood that the various embodiments described herein forusing a variance parameter associated with a device tracker to determineand display an intersection indicator can also be used to determine anddisplay the displayed affected region 206. Furthermore, the varianceparameters described herein can be used alone or in combination with thevarious variance parameters described in the '930 Application todetermine and display the affected regions described therein, includingthe surface display regions.

Various embodiments can include any combinations of the graphicsdescribed above and/or other graphics or image guidance cues. Forexample, in some embodiments, data related to a single surgicalinstrument (such as an ablation needle, ultrasound probe, etc.) can bepresented in more than one manner on a single display. Consider anembodiment in which device 242 is an ablation needle and device 222 isan ultrasound transducer. As mentioned previously, as the medicaldevices are displayed in a virtual 3D space, with the screen 220 actingas a window into the virtual 3D space, if a healthcare provider orientsmedical imaging device 222 such that it is perpendicular to thepoint-of-view or point-of-view location (e.g., perpendicular to thescreen), the perspective view of the ultrasound image 204 would showonly the edge and the contents of the ultrasound image 204 would not bevisible. In some embodiments, the image guidance system can track thehealthcare provider's head using an emplacement sensor and/or a positionsensing unit. In some embodiments, such as, when the head of a user istracked, the healthcare provider can then move her head to the side, sothat she sees the ultrasound image from a different point of viewlocation.

In some embodiments, the image guidance system can concurrently displayan additional 2D view 205 of the ultrasound image, simultaneous to the3D depiction 204, so that the ultrasound image is always visible,regardless of the emplacement in which the healthcare provider holds themedical imaging device 222. The 2D view 205 of the ultrasound data canbe similar to what a healthcare provider is accustomed to seeing withtraditional ultrasound displays. This can be useful to provide thehealthcare provider with imaging to which she is accustomed and allows ahealthcare provider to see the ultrasound data regardless of thethen-current emplacement of the ultrasound probe with respect to theuser.

In some embodiments, the 2D view 205 of an ultrasound image is depictedin the upper right corner of the monitor (though it can be placed in anylocation). In some embodiments, the guidance system can automatically(and continually) choose a corner in which to render the 2D view 205 ofthe ultrasound image, based on the 3D position of the surgicalinstruments in the rendered scene. For example, in FIG. 2, ablationneedle 242 can be held in the healthcare provider's left hand and themedical device shaft is to the left of the 3D view of the ultrasoundimage slice, so that the 2D view 202 of the ultrasound image in theupper right corner of display 220 does not cover any of the 3D featuresof the medical device (or vice-versa). If the medical device were heldin the healthcare provider's right hand (and to the right of theultrasound image 204, the virtual medical device shaft would appear onthe right side. To prevent the 2D view 205 in the corner of display 220from covering the medical device shaft, the system can automaticallymove it to a corner that would not otherwise be occupied by graphics ordata.

In some embodiments, the system 101 attempts to avoid having the 2D view205 of the ultrasound image quickly moving among corners of the displayin order to avoid overlapping with graphics and data in the display. Forexample, a function f can be used to determine which corner is mostsuitable for the 2D ultrasound image to be drawn in. The inputs to f caninclude the locations, in the screen coordinate system, of the displayedmedical device tip, the corners of the 3D view of the ultrasound image,etc. In some embodiments, f s output for any given point in time isindependent of f's output in the previous frames, which can cause theultrasound image to move among corners of the display rapidly. In someembodiments, the image guidance system will filter f's output over time.For example, the output of a filter g, for any given frame, could be thecorner, which has been output by f the most number of times over thelast n frames, possibly weighting the most recent values for f mostheavily. The output of the filter g can be used to determine in whichcorner of display 220 to display the 2D ultrasound image and thetemporal filtering provided by g can allow the 2D view 205 of theultrasound image display to move more smoothly among the corners of thedisplay 220.

In some embodiments, other appropriate virtual information and/or imageguidance cues can be overlaid on the 2D view 205 of the ultrasound imageas well as the 3D view 204. Examples include: orientation indicator 214,an indication of the distance between the medical device's tip and thepoint in the plane of the ultrasound image that is closest to themedical device tip; the cross section or outline of the ablation volumethat intersects with the ultrasound slice; and/or the intersectionpoint, box, outline, etc. between the virtual medical device's axis andthe ultrasound image plane.

Furthermore, it will be understood that other image guidance cues can begenerated and displayed on the display as described in greater detail inthe '274 Application, previously incorporated herein by reference. Forexample, the system 101 can generate and/or display graphical indicatorsthat help indicate the spatial relationship between a medical device andan ultrasound image plane (e.g., graphical image plane indicators) orother plane indicators to indicate the relative positions of the virtualmedical device(s) and ultrasound image, features of interest,annotations, foundational plane indicators, foundational planeintersection indicators, other graphical indicators, approximate medicaldevice location indicators, etc. As described in greater detail aboveand in the '274 Application, the various image guidance cues can begenerated based at least in part on the emplacement information of themedical devices used with the system 101.

Figures Overview

FIGS. 3A-3C, 5A-5C, 6A-6C, 7A-7C, 10A, 10B, 8A, 8B, 9A-9C, 10A, 10B,11A, and 11B are diagrams of embodiments illustrating variousperspective views of scenes including various display objects. Thescenes depicted therein can be displayed as part of the images 150and/or on the screen 220. For example, the scenes depicted in FIGS. 3A,5A, 6A, 7A, 9A 10A, and 11A can represent embodiments illustratingdisplay objects that can be displayed in the 2D viewing area 152 and/orin the 3D viewing area 154 when the medical imaging device correspondingto virtual medical device 302 (and/or its image plane/region) is atleast approximately parallel with respect to a point-of-view-location.The scenes depicted in FIGS. 3B, 5B, 6B, 7B, and 9B can representembodiments illustrating display objects that can be displayed in the 3Dviewing area 154 when the medical imaging device corresponding tovirtual medical device 302 (and/or its image plane/region) is at leastapproximately orthogonal or perpendicular with respect to thepoint-of-view-location. The scenes depicted in FIGS. 3C, 5C, 6C, 7C, 8A,8B, 9C, 10B, and 11B can represent embodiments illustrating displayobjects that can be displayed in the 3D viewing area 154 when themedical imaging device corresponding to virtual medical device 302(and/or its image plane/region) is neither parallel nor orthogonal withrespect to the point-of-view-location. In addition, similarly numbereddisplay objects across the figures correspond to similar displayobjects.

Furthermore, it will be understood that any one or any combination ofthe display objects shown in the figures can be displayed alone or incombination with any one or any combination of the display objects shownin other figures. For example, in some embodiments, only an intersectionindicator or variance volume indicator is displayed on the screen 220.In certain embodiments, only an intersection indicator and at least oneof the virtual medical imaging device or virtual medical device aredisplayed on the screen 220, etc.

Emplacement Variance

In some cases, there can be some uncertainty as to the preciseemplacement of the tracked medical device 242. For example, a devicetracker associated with the medical device can have some error orvariance associated with it (e.g., 5%-10%). The variance may be due tonoise, jitter, vibration, manufacturing and/or mechanical tolerancesinvolved when affixing the pose sensor to the medical devices. Thevariance may result in the precise emplacement of the medical devicebeing different than the emplacement of the virtual medical devicedisplayed on the screen 220.

In some cases, the operating parameters of a device tracker can includeone or more emplacement variance parameters indicating the amount ofvariance that a healthcare provider can expect when using a particulardevice tracker. The emplacement variance parameter may indicate apercentage certainty of the location of the device tracker (non-limitingexamples: 95% or 99%) and/or may indicate that a device tracker operateswithin a certain range, or that a healthcare provider can expect acertain variance in location, such as a particular standard deviationand/or +/− some percent. For example, the emplacement variance parametermay indicate that the location of the device tracker is in a particularlocation with an error of one or more millimeters or 5%-10%, etc.

In some embodiments, the system can account for some uncertainty aboutthe spatial relationship between the virtual medical device and theimage plane/region using the variance parameter. For example, in someembodiments, the system models, the medical device, not as just aone-dimensional axis, but as a three dimensional tube, with a diameterthat is related to the accuracy of the tracking system (accuracy can bebased on measurement error, noise, jitter, vibration, due attributableto the user and/or attributable to the pose sensor). Similarly, incertain embodiments, the system can model the image plane/region as athree-dimensional slab, with a thickness that is related to the accuracyof the system. The system can use the uncertainty to generate anddisplay to the user graphic indicators (analogous to error-bars on ascientific plot), such as a variance volume indicator, intersectionindicator, etc.

FIGS. 3A-3C are diagrams of an embodiment illustrating variousperspective views of a scene including a variance volume. In theillustrated embodiments of FIGS. 3A-3C, the display objects include avirtual medical imaging device 302, an image region (displayed as animage area) 304 that includes an ultrasound image, a virtual medicaldevice 306, a trajectory indicator 308, a variance volume indicator 310,image width indicators 312 a, 312 b, and image slab 314.

It will be understood that any one or any combination of the displayobjects shown in FIGS. 3A-3C can be displayed alone or in combination asdesired. For example, in some embodiments, only the variance volumeindicator 310 is displayed on the screen 220. In certain embodiments,only the variance volume 310 and the virtual medical device 306 aredisplayed on the screen 220, etc.

To determine and display the virtual medical imaging device 302, thesystem 101 can use emplacement data associated with a medical imagingdevice that corresponds to the virtual medical imaging device 302 and/ordimensions of the medical imaging device. The system 101 can use thedimensions to determine the shape and dimensions of the virtual medicalimaging device 302 and/or can use the emplacement data to determine theemplacement of the virtual medical imaging device 302. Using theemplacement data associated with the medical device that corresponds tothe virtual medical device 306 and/or dimensions of the medical imagingdevice, the system 101 can determine and display the virtual medicaldevice 306.

Similarly, the system 101 can use the emplacement data associated withthe medical imaging device and/or its operating parameters to determineand display the image region 304. For example, the system 101 can usethe operating parameters (e.g., height, length, width) to determine, theshape and dimensions of the image region and/or can use the emplacementdata to determine the emplacement of the image region 304. In addition,as imaging data is received from the medical imaging device, the system101 can combine the imaging data with the image region 304, asillustrated in FIGS. 3A, 3B. For example, the system 101 can map theimaging data from one coordinate system to an image region 304coordinate system and/or the display coordinate system. It will beunderstood that although reference throughout is made to displaying theimage region as an image area, the image region can be displayed as animage volume.

Image width indicators 312 a, 312 b and image slab 314 are shown inFIGS. 3B, 3C to illustrate a thickness of the image region/imaged regionthat corresponds to the image data displayed in the image region 304 andmay or may not be displayed as part of the image 150. The image widthindicators 312 a, 312 b can correspond to different planes or differentareas of the image slab 314, image volume, or imaged region. The shapeand dimensions of the image width indicators 312 a, 312 b can beassociated with, or correspond to, the shape of the image region 304and/or the shape of an image generated by the medical device 302. Imagewidth indicator 312 a can correspond to a proximal width (e.g., aportion of the width that is closer to the point-of-view location thanthe image region 304) and image width indicator 312 b can correspond toa distal width (e.g., a portion of the width that is further from thepoint-of-view location than the image region 304). Of course, it will beunderstood that the proximal and distal width indicators can changedepending on the orientation of the medical device/virtual medicaldevice. Image slab 314 can correspond to the image region and/or theimage region between the image width indicators 312 a, 312 b and caninclude the image region 304.

As mentioned above, medical imaging devices often capture across-section of tissue having a certain thickness. However, in someinstances, a medical imaging device treats the captured cross-section asif there is no thickness and compacts the image data so that there is noperceived thickness in the image data. The image width indicators 312 a,312 b and/or image slab 314 can be used to illustrate a width of theimaged region that corresponds to the image data that is shown in theimage region 304, prior to the image data being compressed. In addition,the image width indicators 312 a, 312 b and/or image slab 314 can beused to determine one or more intersection indicators as described ingreater detail below with reference to FIGS. 6A-6C and 7A-7C. Similar tothe image region 304, the image width indicators 312 a, 312 b and/orimage slab 314 can be determined and displayed based at least in part onthe emplacement data and/or the operating parameters associated with thevirtual medical imaging device 302. It will be understood that althoughthe image width indicators 312 a, 312 b in the illustrated embodimentcorrespond to the ends of the image region, that any plane or planes ofthe image slab 314 or image region can be used to determine and displaythe image indicator 710.

To determine the variance volume, the system 101 can use a determinedaxis of the virtual medical device 306 and the variance parameters. Forexample, the system 101 can calculate a volume surrounding the axisbased at least in part on the variance parameter. Once determined, thesystem 101 can cause a variance volume indicator 310 to be displayed onthe screen 220.

In some embodiments, to display the variance volume indicator 310, thesystem 101 can determine the emplacement of the virtual medical devicebased at least in part on the emplacement data associated with a medicaldevice and display variance volume indicator based at least in part onthe determined emplacement of the virtual medical device. In some casesdetermining the emplacement of the virtual medical device can includereceiving x, y, z and/or quaternion orientation coordinates associatedwith a device tracker that is associated with the medical device andmapping the received coordinates to display coordinates associated withthe screen 220. Once the emplacement of the virtual medical device isdetermined, the system 101 can determine the variance volume as a volumesurrounding the medical device and/or the axis of the medical device. Todisplay the variance volume indicator 310 (and/or any display objectdescribed herein), the system 101 can communicate the coordinates of thevariance volume (and/or any display object described herein) to agraphics rendering engine, which can convert the emplacement data todisplay coordinates and communicate the resulting rasterized image to adisplay buffer, which is then converted to a video signal (e.g.DVI/HDMI) and sent to a display screen 220.

In the illustrated embodiment, the variance volume indicator 310 isdisplayed using multiple, equally-spaced rings surrounding and along theaxis of the virtual medical, as well as two lines illustrating theperimeter of the variance volume. In this way, the variance volumeindicator 310 appears to be made up of multiple cylinders stacked on topof one another. However, it will be understood that the variance volumeindicator 310 can be displayed in a variety of ways, as desired. Forexample, the rings may not be equally-spaced or not included at all.Similarly, the perimeter lines may be omitted. In some embodiments, thevariance volume indicator 310 can be displayed as a long tube along theaxis of the virtual medical device. The tube can be shaded or texturedas desired.

The variance volume can also be drawn as animated bars that rotate aboutthe center of the medical device axis. In some embodiments, thelongitudinal axis of the bars can be parallel to the longitudinal axisof the medical device. In certain embodiments, the longitudinal axis ofthe bars can be at an angle with respect to the longitudinal axis of themedical device (e.g., perpendicular, forty-five degrees, or any otherangle). Each bar can be a predetermined length that spans the entiremedical device or just a portion thereof. Furthermore, there can bemultiple sets of rotating bars at different locations along the medicaldevice. Each set of rotating bars along the axis can include one or morebars. For example, in some embodiments, there can be three sets (eachset located at a different position along the medical device) of threebars each. It will be understood that any number of sets of bars can beused and that each set can include any number of bars.

In some embodiments, the variance parameter can be fixed such that thevariance volume does not change. In certain embodiments, the varianceparameter may change over time. In such embodiments, the variance volumeand the variance volume indicator can change over time as well. In someembodiments, the variance parameter may not be uniform for the length ofthe needle. In such embodiments, the variance volume indicator 110 canindicate the different variance parameters along the medical device.Furthermore, although the variance parameter has been described withrespect to the virtual medical device 306, it will be understood that avariance parameter associated with the virtual medical device 302 can beused alone or in combination with the variance parameter associated withthe virtual medical device 306. The variance parameter associated withthe virtual medical device 302 can be used to determine the emplacementof the virtual medical device 302 and medical device associatedtherewith and to display a variance volume associated with the virtualmedical device 302. Accordingly, more than one variance parameter can beused by the system 101 to determine emplacement of the virtual medicaldevices 302, 206 and display them on the screen 220.

FIG. 4 is a flow diagram illustrative of an embodiment of a routine 400implemented by the system 101 to display at least a portion of avariance volume or a variance volume indicator. One skilled in therelevant art will appreciate that the elements outlined for routine 400can be implemented by one or more computing devices/components that areassociated with the system 101, such as the position sensing unit 106,the image guidance unit 104, surgical system 108, and/or the imager 110.Accordingly, routine 400 has been logically associated as beinggenerally performed by the system 101. However, the followingillustrative embodiment should not be construed as limiting.Furthermore, it will be understood that the various blocks describedherein with reference to FIG. 4 can be implemented in a variety oforders. For example, the system may implement some blocks concurrentlyor change the order, as desired.

At block 402, the system 101 receives emplacement data associated with amedical device. In some embodiments, the emplacement data can bereceived from a device tracker associated with the medical device and/ora position sensing unit 106. In certain embodiments, the emplacementdata includes emplacement coordinates, such as, but not limited to oneor more x, y, z position coordinates and/or quaternion orientationcoordinates.

At block 404, the system 101 determines emplacement of a virtual medicaldevice associated with the medical device. In some embodiments, thesystem 101 uses the emplacement data associated with the medical deviceto determine the emplacement of the virtual medical device. In certainembodiments, the system 101 determines the emplacement by converting theemplacement coordinates corresponding to a coordinate system of theposition sensing unit 106 to a coordinate system corresponding to adisplay. In some embodiments, the system 101 can use dimensions of themedical device (e.g., length, circumference, etc.) to determine theemplacement of the virtual medical device. The system 101 can include anon-transitory computer-readable medium that stores the dimensions ofvarious medical devices that can be used with the system and/or receivethe dimensions from the medical device or elsewhere dynamically.

At block 406, the system 101 can determine a variance volume based atleast in part on one or more operating parameters associated with thedevice tracker that is associated with the medical device. In someembodiments, the operating parameters can include a location varianceparameter. In some embodiments, the system 101 can use a determined axisof the virtual medical device and the variance parameter to determinethe variance volume. In certain embodiments, the system 101 can use thetrajectory of the virtual medical device and/or the dimensions of thevirtual medical device to determine the variance volume.

At block 408, the system 101 can cause one or more displays to display avariance volume indicator or at least a portion of the variance volume,as described in greater detail above. In some embodiments, the variancevolume indicator can correspond to at least a portion of the variancevolume that intersects with an image region, as will be described ingreater detail below with reference to FIGS. 5A-5C. In such embodiments,the variance volume indicator can also be referred to as an intersectionindicator.

In some embodiments, to cause the one or more displays to display thevariance volume indicator (and/or any of the display objects describedherein), the system 101 can communicate the emplacement data of thevariance volume (and/or display object) to graphics rendering engine,which can convert the emplacement data to display coordinates andcommunicate the resulting rasterized image to a display buffer, which isthen converted to a video signal (e.g. DVI/HDMI) and sent to a displayscreen 220.

It will be understood that fewer, more, or different blocks can be usedas part of the routine 400. For example, in some embodiments, the system101 can cause the one or more displays to display the virtual medicaldevice as illustrated at block 410 and described in greater detailabove. Furthermore, the blocks of routine 400 can be combined with anyone or more of the blocks described below with reference to FIGS. 12 and13.

Intersection Indicator

The emplacement variance can also affect the certainty with regard tothe emplacement of an intersection between the imaged region and themedical device. FIGS. 5A-5C are diagrams of an embodiment illustratingvarious perspective views of a scene including display objects. In theillustrated embodiments of FIGS. 5A-5C, the display objects include thevirtual medical imaging device 302, an image region 304 that includes amedical image, a virtual medical device 306, a trajectory indicator 308,an intersection indicator 510, image width indicators 312 a, 312 b, andimage slab/region indicator 314.

To determine the intersection indicator 510, the system 101 can use theemplacement data associated with the medical device and the medicalimaging device. The system can determine the intersection in the displaycoordinate system and/or the coordinate system of the position sensingunit 106. For example, the system 101 can determine an intersection ofthe trajectory and/or axis of the medical device and the imageplane/region and/or the imaged region and/or an intersection of thetrajectory and/or axis of the virtual medical device with the imageplane/region. In addition, the system can determine the intersection ofan axis associated with the first emplacement data, or first emplacementdata axis, and a plane associated with the second emplacement data, orsecond emplacement data plane. For simplicity, reference is made todetermining an intersection of the image plane/region and the virtualmedical device 302, however, it will be understood that other methodscan be used to determine an intersection of the image plane/region orimaged region and an axis of the medical device.

In some embodiments, to determine the intersection, the system 101 cancompare the various coordinates of the two objects (e.g.,axis/trajectory and region/plane). If a pair of coordinates (e.g., thex, y, z coordinates from each object) match (e.g., are equal) or satisfya distance threshold, the system can determine that the two objectsintersect. In certain embodiments, the system 101 can determine thatthere is an intersection if the trajectory/axis of the virtual medicaldevice 302 and a portion of the image plane/region can be mapped to thesame pixels in a video or image output data buffer.

The distance threshold can be a predefined distance, such as one or morebits, one or more pixels, etc. In some embodiments, the distancethreshold can be based at least in part on whether the distance betweenthe coordinates is perceptible to a user, which may be based at least inpart on the size of the display, the size of the display relative to theimage and/or imaged region, and/or the distance between thepoint-of-view location and the display, etc. For example, in some cases,the distance threshold can be smaller for larger displays (or largerdisplay:image ratios) and larger for smaller displays (or smallerdisplay:image ratios), or vice versa. In certain cases, the distancethreshold can be larger for larger distances between the point-of-viewlocation and the display and smaller for smaller distances between thepoint-of-view location and the display, or vice versa. In certainembodiments, the distance threshold can be different for eachcoordinate.

In certain embodiments, the system 101 can perform the comparison foreach location along the axis of the virtual medical device and/or imageplane/region. In some cases, the system can determine that there is anintersection if the axis/trajectory of the virtual medical device and aportion of the image plane/region are level and have the same depth.

As mentioned above, any coordinate system can be used to compare thecoordinates of the virtual medical device with the image plane/regionand/or to determine whether the virtual medical device and the imageplane/region intersect. For example, the coordinate system of thedisplay and/or the coordinate system of position sensing unit 106 can beused as desired.

In some embodiments, for each location on the display, the system canquery whether a portion of the two display objects have been (or willbe) mapped to that location. If the system 101 determines that a portionof two display objects (e.g., the trajectory of the virtual medicaldevice and the image region have been (or will be) mapped to thatlocation, the system 101 can determine that the two display objectsintersect. In certain embodiments, the system 101 can determine that thetwo display objects satisfy the location threshold and/or intersect ifthe two objects map to the same location on a display, such as the samepixel or same array of pixels.

In some embodiments, the system 101 can determine the intersectionindicator based at least in part on the determined intersection of theaxis of the virtual medical device and the image region and a varianceparameter associated with the device tracker associated with the virtualmedical device. For example, once the intersection is determined, thesystem 101 can use the variance parameter to generate an area on theimage region 304 that represents the potential location of intersection.In some cases, the center of the area can correspond to the determinedpoint of intersection.

In some embodiments, the system 101 can determine and/or display the 3Dshape that results from the intersection of the slab containing theimage plane (e.g., the volume between the image width indicators 312 a,312 b), and the variance volume containing the medical device axis(illustrated as a cylinder surrounding the medical device and itstrajectory, with multiple circles at regular (or irregular) intervals).This can be drawn, transparently composited, with the image and medicaldevice or medical device trajectory.

In certain embodiments, the system 101 can use the intersection of adetermined variance volume with the image plane/region to determine theintersection indicator. For example, the area or perimeter of thevariance volume that intersects with the image plane/region can be usedas the intersection indicator. Such an embodiment is illustrated inFIGS. 5A and 5C. In the illustrated embodiment, the shape of theintersection indicator 510 is ellipsoid, however, it will be understoodthat any shape can be used as desired. For example, the shape can berectilinear, as shown and described in greater detail below withreference to FIGS. 9A-9C and/or can include two parallel line segmentsconnected by semi-circular end caps (obround shape), as shown anddescribed in greater detail below with reference to FIGS. 6A-6C and7A-7C.

Orientation Angle

In some embodiments, the system 101 can use an orientation angle orangle-of-approach to determine and display the intersection indicator510. The orientation angle or angle-of-approach can correspond to theangle between the image plane and the medical device axis and/or thevirtual medical device axis.

As the angle-of-approach decreases (e.g., the medical device is movedcloser to being parallel with the image plane), the potential point(s)of intersection increase. Accordingly, in some embodiments, the system101 can use the angle-of-approach to determine and display theintersection indicator. In certain embodiments, when the orientationangle satisfies a threshold angle, the intersection indicator can bedeactivated.

In some embodiments, the system 101 can generate a 3D intersection shapedescribed above and/or generate an intersection shape based on theintersection of the variance volume and the image plane/region, projectit onto the image plane, and draw its outline on the image plane. Forexample, if the medical device (or virtual medical device) axis is at a90-degree angle-of-approach to the image plane, the system can draw ahollow circle. As illustrated in FIGS. 5A-5C, as the angle-of-approachis reduced, the circle can become an ellipse. The minor diameter of theellipse can correspond to the circle's (related to the accuracy of thesystem), whereas the major diameter can be related to theangle-of-approach. As the angle becomes shallower, the ellipse's majordiameter can become greater. Furthermore, some of the various figuresincluded herein show differently sized intersection indicators, which,in some, embodiments, can be due to a different angle-of-approach.

Additionally, when the angle-of-approach becomes zero (or approximatelyzero), the major axis becomes infinite, and the intersection indicatorcan become two parallel lines, which can be unbounded. A non-limitingexample is shown in FIGS. 11A and 11 b, in which the intersectionindicator 1110 is shown as two parallel lines. A non-limiting benefit ofthis embodiment is that it visually obscures less of the image regionthan a 3D intersection-indicator shape. In addition, when the clinicianis approaching in-plane, or at shallow angles-of-approach, the parallellines do not jump or move erratically, and the clinician can use it bymanipulating the medical device and U/S probe such that the target is inbetween the two parallel lines, and, in this arrangement, she candetermine that the medical device will hit the target if she maintainsthis orientation while driving the medical device forward into thetissue. This intersection-indicator smoothly can transition from twoparallel lines to a circle, as the angle-of-approach changes.

In some embodiments, the system 101 can compare the orientation of theaxis of the medical device or virtual medical device with theorientation of the image plane/region. Based on the comparison, thesystem 101 can increase or decrease the size of the intersectionindicator. In some embodiments, as the angle between the orientationsdecreases (moves towards parallel), the size of the intersectionindicator can increase. In certain embodiments, when the angle betweenthe orientations is equal or approximately equal, the intersectionindicator can be displayed as two parallel lines, which can be unboundedparallel lines.

Obround Intersection Indicator

FIGS. 6A-6C are diagrams of an embodiment illustrating variousperspective views of a scene including a trajectory indicator having anobround shape. As shown in FIGS. 6A-6C, the shape of the intersectionindicator 610 can be obround. The shape can be achieved by displaying arounded edge on two sides with two parallel lines connecting the roundededges. In some embodiments, the length of the parallel line segments cancorrespond to the angle-of-approach and/or with the major diameter ofthe elliptical intersection-indicator mentioned above. By using paralleland circular line segments (projected onto the image plane), the systemcan make it easier for the user to understand the angle between theimage plane and the display surface of the screen 220 (via theperspective foreshortening effect).

In some embodiments, the system can use the intersection of the axis ofthe virtual medical device with multiple planes of the image region orimage slab to determine and display the intersection indicator. FIGS.7A-7C are diagrams illustrating an embodiment for determining anddisplaying a trajectory indicator using the image region or slab. Asmentioned above, in some embodiments, the system 101 can generate a 3Dintersection shape described above and/or generate an intersection shapebased on the intersection of the variance volume and the image region(or image slab), project it onto the image plane, and draw it (or itsoutline) on the image plane as the intersection indicator 710.

The intersection indicator 710 can be determined in a variety of waysusing the image region/slab 314. In some embodiments, the intersectionshape can be based on the intersection of the variance volume across theimage region or slab 314 and in certain embodiments, the intersectionshape can be determined based on the intersection of the variance volumewith one or more planes within the image region/slab 314.

For example, in some cases, the intersection indicator 710 can be basedat least in part on the intersection of the variance volume with theimage width indicator 312 a (displayed as intersection indicator 712 a),with the image width indicator and 312 b (displayed as intersectionindicator 712 b), and with the image region 304 (displayed asintersection indicator 712 c). Combining or projecting the threeintersection indicators 712 a, 712 b, 712 c onto the image region 304can result in the intersection indicator 710 displayed on image region304. In some cases, as part of the projecting, the rounded portions ofthe indicators 712 a, 712 b, 712 c between the distal rounded portionscan be removed.

In some embodiments, the intersection indicator 710 can be based on onlythe intersection indicators 712 a and 712 b and/or can omit determiningthe intersection of the variance volume with the image plane 304. Insuch embodiments, when the intersection indicators 712 a and 712 b areprojected onto the image region 304, lines can be drawn between thedistal rounded edges to generate the intersection indicator 710. Asmentioned above, in some cases, as part of the projecting, the roundedportions of the indicators 712 a, 712 b between the distal roundedportions can be removed.

Device-Blocking-Objects

FIGS. 8A and 8B are diagrams of embodiments illustrating an intersectionindicator for a scene intersecting with device-blocking display objects.In addition to determining intersections between an axis of the medicaldevice and the image plane/region and displaying associated intersectionindicators, the system 101 can also determine and display theintersection indicator 810 a, 810 b when the axis intersects with otherdisplay objects. For example, in some situations, the trajectory of themedical device 306, on its way to the image plane, might be blocked byobjects or 3D structures that the clinician wishes to avoid hitting.Examples include the housing of the medical device associated with thevirtual medical device 302, blood vessel 802, nerves, ribs or otherbones, etc., which can be referred to generally asdevice-blocking-objects. FIG. 8A is a diagram illustrating an embodimentof an intersection indicator being 810 a displayed on portions of thevirtual medical device 302 and FIG. 8B is a diagram illustrating anembodiment of an intersection indicator 810 b being displayed onportions of a blood vessel 802.

Based on emplacement data associated with these device-blocking-objects,the system can display corresponding display objects in their properspatial arrangement with the image plane and the virtual medical device.The emplacement data can be received from device trackers, positionsensing units 106, and/or based on 3D image data, such as a registeredCT, MRI, etc. Using the emplacement data associated thedevice-blocking-objects and the medical device, the system 101 candetermine any intersections. Based on the determined intersections, thesystem 101 can project the intersection indicator onto the surfaces ofthe device-blocking-blocking display objects. In some embodiments, thesystem 101 can determine which objects are device-blocking-objects bydetermining that an object intersects with the axis or variation volumeof the medical device and the intersection is located between themedical device and an intersection of the medical device and the imageregion.

In addition, portions of the intersection indicator on the image planethat are blocked by a device-blocking object can be a different color,grayed out, or not drawn. For example, portions of the intersectionindicator 810 a that are blocks by the blood vessel 802 are displayeddifferently than portions of the intersection indicator 810 a that arenot blocked by the blood vessel 802. In this way, the system 101 canconvey to a user that the device will not reach the target seen in theimage region (if the medical device trajectory is not changed), whichdevice-blocking-objects are in the way, which portions of them are inthe path of the medical device, and if the medical device trajectory isbarely or nearly blocked (since the intersection indicator has somewidth, diameter related to the accuracy of the system).

With respect to FIG. 8A, the system 101 can determine that thetrajectory (and/or variance volume) of the medical device associatedwith the virtual medical device 306 intersects with a portion of theblood vessel 802. The emplacement data for the blood vessel can be basedon an emplacement registered 3D image, such as a registered CT scan,MRI, CAT scan, etc. Based on the determined intersection, the system 101can cause a display to display at least a portion of the intersectionindicator onto the blood vessel 802. To draw the intersection indicator810 a onto the blood vessel, the system 101 can rely on the emplacementdata of the blood vessel.

FIG. 8A further illustrates a shaded region 804 to indicate how portionsof the image region 304 may not be accessible without going through thedevice-blocking-objects. In the illustrated embodiment, using theemplacement data of the image region 304 and the emplacement data of theblood vessel 802, the system 101 can determine a shaded region 804 basedon which portions of the image region 304 are blocked by the bloodvessel 802. The system can then display the shaded region 804 toindicate to a user the portions of the image region 304 that are notaccessible based on the current position and/or orientation of themedical device. For example, the shaded region 804 can indicate whatportions of the image region 304 are not accessible so long as thecurrent position of the medical device is maintained and/or even if thecurrent orientation of the medical device is adjusted.

Similar to FIG. 8A, FIG. 8B illustrates an embodiment in which portionsof the image region 304 are blocked by a device-blocking-object. In theillustrated embodiment of FIG. 8B, the device-blocking-object is thevirtual medical device 302. Based on a determined intersection of theaxis and/or the variance volume of the virtual medical device 306 withthe virtual medical imaging device 302, the system 101 can display aportion of the intersection indicator 810 a, 810 b on the virtualmedical imaging device 302. In addition, based on a determination that aportion of the image region 304 is not accessible based on the currentposition and orientation of the medical device with respect to the imageregion, the system 101 can display a shaded region 804 that correspondsto the portions of the image region 304 that are not accessible.

FIGS. 8A and 8B further illustrate that, in some embodiments, the sizeand shape of the intersection indicators 810 a, 810 b can vary based atleast in part on the emplacement of the virtual medical device 206 withrespect to the image region. For example, as described in greater detailabove, the size of the intersection indicator can change in relation toa changed orientation angle between the medical device and image region.In the illustrated embodiments of FIGS. 8A and 8B, the intersectionindicator 810 a is larger and more elongate than the intersectionindicator 810 b, which can be due in part to a smaller orientation anglebetween the medical device and image region in the embodimentillustrated in FIG. 8A compared to the orientation angle in theembodiment illustrated in FIG. 8B.

FIGS. 9A-9C are diagrams of an embodiment illustrating variousperspective views of a scene including multiple intersection indicators910 a, 910 b corresponding to different medical devices 306 a, 306 b. Inthe illustrated embodiment, the intersection indicators 910 a, 910 b areillustrated as rectilinear. In some embodiments, the corners of theintersection indicators can be drawn such that they have one or moreline segments which are parallel to the edges of the image region 304.In this way, a user can more easily distinguish the intersectionindicators 910 a, 910 b from other display objects, and can understandthat the intersection indicator exists in the same plane as the imageregion 304.

The rectilinear display objects 910 a, 910 b can be determined similarto the determination of the intersection indicators described above withreference to FIGS. 5A-5C, 6A-6C, 7A-7C, 8A, and 8B. For example, therectilinear intersection indicators can be determined based at least inpart on an intersection of a first emplacement data axis with at least aportion of an image region and/or other display objects. In theillustrated embodiment, the system determines that the axis of thevirtual medical device 306 a intersects with the virtual medical device302 and at least a portion of the image region, and the intersectionindicator 910 a is drawn on the virtual medical device 302 and in theimage region 304. In addition, in the illustrated embodiment, the systemdetermines that the axis of the virtual medical device 306 b intersectswith at least a portion of the image region and the image indicator 910b is displayed in the image region 304.

Image Region and Medical Device Image Difference

FIGS. 10A, 10B are diagrams of an embodiment illustrating variousperspective views of a scene including multiple display objects. Asmentioned previously, any one or any combination of the embodimentsdescribed herein can be displayed concurrently. FIGS. 10A, 10B depictembodiments in which many of the display objects described above withrespect to FIGS. 3A-3C, 5A-5C, 6A-6C, 7A-7C, 8A, and 8B, areillustrated. For example, FIGS. 10A, 10B illustrate a virtual medicalimaging device 302, an image region 304 displayed as an image area thatincludes a medical image 1002, a virtual medical device 306, atrajectory indicator 308, a variance volume indicator 310, a shadedregion 804, and an intersection indicator 1010 intersecting with thevirtual medical imaging device 302 and the image region 304.

In some embodiments, the medical image to be displayed on the imageregion may not match the size and/or shape of the image region 304, asillustrated in FIGS. 10A, 10B. For example, the image region 304 may berectangular and the medical image 1002 may not be rectangular (e.g. itcan be a convex scan and/or trapezoidal) and/or the image region 304 maybe larger than the medical image 1002. In such embodiments, the systemcan bound the medical image 1002 with the image region 304 and/or orsuperimpose a grid or grid points over the medical image 1002 and/orimage region 304, to help the user understand the orientation of themedical image and/or image region in the 3D context. In certainembodiments, the area outside the medical image 1002 and inside theimage region 304 can be colored (e.g., black) differently from themedical image 1002 to distinguish it from the medical image 1002.

In some embodiments, the system 101 can compare the dimensions of themedical image 1002 with the dimensions of the image region 304. Incertain embodiments, the dimensions can be stored in a non-transitorycomputer-readable medium associated with the system 101 and/or can bereceived from the medical imaging device, etc. Upon determining that thecompared dimensions are different, the system 101 can identify theportions of the image region 304 that are outside, or do not overlapwith, the medical image 1002. When displaying the image region 304 andthe medical image 1002, the system 101 can display the portions of theimage region 304 that overlap with the medical image 1002 with themedical image 1002 and can display the portions of the image region 304outside, or that do not overlap with, the medical image 1002 differentlyto highlight the difference from the medical image 1002.

Off-Image Region Intersection

FIGS. 11A and 11B are diagrams illustrating an embodiment in which anintersection indicator is located outside the image region. In someembodiments, the system 101 can determine that the intersectionindicator would be located outside of the image region (e.g., theintersection of the axis or variance volume of the medical device andthe image plane is outside the image region) and/or outside the regionof the display screen. When this happens, the system may draw one ormore visual elements that indicate to the user, that the intersectionindicator is outside the image region and/or off-screen.

The visual elements can include, as non-limiting examples, any one orany combination of changing the border of the image region 304 to bethicker 1102, and/or a different color (e.g. add a thick red border);altering the display of the image region 304 (increasing brightness,contrast); altering the display of the entire scene (e.g., displaying aborder around the scene or changing the background of the displayobjects, etc; displaying a warning sign; displaying an arrow 1104indicating in which direction the intersection indicator 1110 lies(e.g., the arrow 1104 may be overlaid/superimposed onto the image region304, or drawn just beyond the border of the image region 304);displaying the numerical distance between the intersection indicator andthe image region 304 (e.g., measured from the center of the image region304, from a border of the image region 304, or other location); and/ordisplaying one or more lines connecting the intersection indicator to aportion of the image region's border (line 1106) or to the center of theimage region 304 (line 1108), to visually indicate theintersection-indicator's location to the user, as also described in U.S.application Ser. No. 12/703,118, incorporated herein in its entirety.

The connecting lines 1106, 1108 can, in some embodiments, include adistance indicator, such as a numerical display, or ruler tick marks. Insome cases, the distance indicator can be superimposed on the connectinglines 1106, 1108, to show how far the intersection indicator 1110 isfrom the image region 304, and to show if it is moving toward or awayfrom the image region 304. In some embodiments, the ruler's 0-mark canbe anchored at the center of the intersection indicator 1110. In thisway, even if the intersection indicator 1110 is off-screen, the user cansee if it is moving closer to the image region 304 (which can be visibleon-screen) because the ruler tick marks will appear to scroll toward theimage region. Similarly, if the intersection-indicator is moving fartheraway from the image region 304, the tick marks can appear to scroll awayfrom the image region 304.

In some embodiments the connecting line 1108 can connect to the centerpoint (or some other point) of the intersection indicator 1110. However,when the medical device is almost parallel to the image plane (i.e. theangle between medical device and the image plane approaches zerodegrees), the intersection indictor 1110 can fluctuate significantly.Accordingly, in certain embodiments, the connecting line 1108 can beconstructed as follows: line 1106 can be determined from the center ofthe image region 304 to the point on the intersection indicator 1110that is closest to the center of the image region 304. Where line 1106intersects the edge of the image region 304, the connecting line 1108can begin, and can continue until it reaches the intersection indicator1110. In certain embodiments, the connecting line 1108 can be alignedsuch that it is perpendicular to the closest edge of the image region304.

In some embodiments, the system can determine the emplacement of themedical device and the image region. In addition, the system candetermine an axis and/or variance volume associated with the medicaldevice does not intersect with the image plane within the image regionand/or does not intersect with the image plane within the boundaries ofthe display screen. Based at least in part on this determination, thesystem can determine and display one or more visual indicatorsindicating that the axis/variance volume does not intersect with theimage region 304, as described in greater detail above. In some cases inwhich the system 101 determines that the axis/variance volume intersectswith the image plane outside the image region but within the displayscreen, the system 101 can display the intersection indicator 1110outside the image region 304.

As described previously, the system can use an internal model for thelength of the needle to determine whether the needle can reach aparticular location inside the patient. For example, trajectory rings,similar to those described in greater detail in U.S. application Ser.No. 12/703,118, can extend forward of the tip, and be projected somedistance forward of the tip, such as the needle length (distance fromneedle tip to needle handle), to allow the clinician to see when theneedle is not long enough to reach the target. In certain embodiments,the system can display the intersection indicator in a different style(e.g. change the color, line width, symbol, line dashes, etc.) whendistance from the intersection indicator to the medical device tip isgreater than the length of the needle. This allows the user to knowwhere the needle is aimed, but also determine that the needle cannotreach a particular location. Accordingly, in some embodiments, thesystem can determine a trajectory of the medical device and cause one ormore displays to display trajectory indicators based at least in part onthe length of the medical device.

FIG. 12 is a flow diagram illustrative of an embodiment of a routine1200 implemented by the system 101 to display at least an intersectionindicator. One skilled in the relevant art will appreciate that theelements outlined for routine 1200 can be implemented by one or morecomputing devices/components that are associated with the system 101,such as the position sensing unit 106, the image guidance unit 104,surgical system 108, and/or the imager 110. Accordingly, routine 1200has been logically associated as being generally performed by the system101. However, the following illustrative embodiment should not beconstrued as limiting. Furthermore, it will be understood that thevarious blocks described herein with reference to FIG. 12 can beimplemented in a variety of orders. For example, the system mayimplement some blocks concurrently or change the order, as desired.

At block 1202, the system receives first emplacement data associatedwith a first medical device. At block 1204, the system receives secondemplacement data associated with a second medical device. In certainembodiments, one of the medical devices can be a medical imaging device,as discussed in greater detail above. Furthermore, as mentioned above,the emplacement data associated with each medical device can be receivedfrom one or more device trackers associated with the respective medicaldevice and/or from a position sensing unit that tracks the devicetrackers. In some embodiments, the emplacement data can include 3Dcoordinates and/or quaternion orientation coordinates.

At block 1206, the system determines an emplacement of a first virtualmedical device corresponding to the first medical device based at leastin part on the first emplacement data. In some embodiments, to determinethe emplacement of the first virtual medical device, the system can mapor transform the received emplacement data to a coordinate systemassociated with a virtual 3D space and/or the display screen.Furthermore, the system can use the characteristics (e.g., length,width, shape, location of tip, angle, location of the device tracker onthe medical device, etc.) of the medical device to determine theemplacement of the virtual medical device. In some embodiments, thesystem can use a CAD file, 3D model, or other file to determine thecharacteristics of the medical device and to determine its emplacement.

In some embodiments, the virtual 3D space coordinate system and/or thedisplay screen coordinate system can be based at least in part on apoint-of-view location. Thus, the emplacement of the first virtualmedical device (and other display objects) can be with respect to apoint-of-view location). The point-of-view location can be a fixedlocation with respect to the display screen and/or can be a dynamiclocation. In some embodiments, when the point-of-view location is afixed location it can correspond to a location that does not changeduring a medical procedure. For example, the fixed location cancorrespond to an expected location of a medical practitioner (e.g.,presume that the medical practitioner will be 10 ft. in front of thedisplay screen). In certain embodiments, the dynamic location cancorrespond to a location that can change during a medical procedure. Forexample, the system can track a medical practitioner during a medicalprocedure (e.g., by affixing a device tracker on the medicalpractitioner's head) and adjust the point-of-view location based atleast in part on a real-time determination of the location of themedical practitioner.

At block 1208, the system determines an emplacement of a second virtualmedical device corresponding to the second medical device based at leastin part on the second emplacement data. The emplacement of the secondvirtual medical device can be determined in similar fashion to theemplacement of the first virtual medical device. For example, the systemcan map or transform the emplacement data to a coordinate systemassociated with the virtual 3D space and/or the display screen. Inaddition, the system can use the characteristics of the second medicaldevice to determine its emplacement.

At block 1210, the system determines an emplacement of an image regionbased at least in part on the second emplacement data. In someembodiments, the system can use the characteristics of the secondmedical device to determine the location of the image region associatedwith the second medical device. For example, the characteristics mayindicate a location of an imager on the medical device, as well as thespecification of the image that is generated using the imager (e.g.,height, width, and/or depth). Using this information, the system candetermine the emplacement of the image region associated with the secondmedical device. Furthermore, the system can determine the emplacement ofthe image region directly from the second emplacement data and/or fromthe determined emplacement of the second virtual medical device. Forexample, the system can use a known relationship between the secondemplacement data and the location of the image region (e.g., the imageregion begins 2 in. away from the of the second emplacement datalocation in a particular direction and ends 5 in. away) and/or use aknown relationship between the emplacement of the second virtual medicaldevice and the location of the image region (e.g., the image regionbegins 4 inches from the tip of the virtual medical device and ends atthe tip of the virtual medical device).

At block 1212, the system determines an intersection. In certain cases,the intersection can correspond to an intersection of an axis associatedwith the first emplacement data and any display object (e.g., virtualmedical device, image plane/region, other medical image data (e.g., CTscan, MM scan data), etc.). In some cases, the intersection cancorrespond to an intersection of an axis associated with the firstemplacement data, or first emplacement data axis, and a plane associatedwith the second emplacement data, or second emplacement data plane. Insome embodiments, the first emplacement data axis corresponds to an axisof the first medical device and/or the first virtual medical device andthe second emplacement data plane corresponds to a plane associated withthe second medical device and/or the second virtual medical device.

As mentioned above, using the first emplacement data, the system candetermine a trajectory and/or axis associated with the first medicaldevice (e.g., the first emplacement data axis). Similarly, the systemcan determine a plane associated with the second medical device usingthe second emplacement data (e.g., second emplacement data plane, imageplane/region). In some embodiments, the determined plane can be parallelwith a longitudinal axis of the second medical device and/or the secondemplacement data plane. Using the first emplacement data axis and thesecond emplacement data plane, the system can determine an intersection.In some embodiments, the system can determine an intersection based atleast in part on a determination that two objects are co-located asdescribed in greater detail above.

At block 1214, the system determines an intersection indicator based atleast in part on the determined intersection, a variance parameterassociated with a device tracker that is associated with the firstmedical device, and/or an angle-of-approach. As mentioned above, in someembodiments, the first and/or second emplacement data can include someinaccuracy. Accordingly, the system can determine an intersectionindicator taking into account the inaccuracy by using one or morevariance parameters that are associated with the inaccuracy and/or theangle-of-approach.

In some embodiments, the system can determine the intersection indicatorusing an angle-of-approach, which can correspond to an angle differencebetween the first emplacement data axis and the second emplacement dataplane (e.g., angle between the trajectory of the first medical deviceand the image plane/region). In certain embodiments, the size of theintersection indicator varies in proportion to the angle-of-approach.For example, as the angle-of-approach decreases, the size of theintersection indicator can increase.

In certain embodiments, the system can use an intersection of a variancevolume associated with the first medical device (and/or the firstemplacement data axis) with the image plane/region (and/or secondemplacement data plane or axis) to determine the intersection indicator.For example, the intersection indicator can correspond to an area ofintersection, the outline of the intersection on the image/plane/region,and/or the projection onto the image region of the intersections of thevariance volume with portions of the image region. In some cases, thesystem can use an intersection of the variance volume associated withthe first medical device with image width indicators (or any portions ofthe image region) to determine the intersection indicator. In someembodiments, the system can determine the intersection of the firstemplacement data axis with the second emplacement data plane andgenerate an intersection indicator by determining an area or perimeteraround the determined intersection. The area/perimeter can be determinedusing the variance parameter and/or the angle-of-approach.

At block 1216, the system causes one or more displays to concurrentlydisplay a perspective view of the first virtual medical device based atleast in part on the determined emplacement of the first virtual medicaldevice, a perspective view of the second virtual medical device based atleast in part on the determined emplacement of the second virtualmedical device, a perspective view of the image region based at least inpart on the determined emplacement of the image region, and theintersection indicator. It will be understood that the system 101 canmap any images received from the second medical device and/or image datacorresponding to the determined emplacement of the second medical deviceto the image region and/or display the mapped data as part of the imageregion.

As mentioned above, the display screen coordinate system can correspondto a point-of-view location and the various display objects (virtualmedical devices, image region, intersection indicator, and otherguidance cues) and their emplacements can be determined with respect tothe point-of-view location. Accordingly, the system can cause one ormore displays to display the perspective view of the display objects.

It will be understood that fewer, more, or different blocks can be usedas part of the routine 1200. For example, the routine 1200 can includeblocks for determining and displaying intersection indicators on one ormore display objects other than the image region and/or a shaded region,as described in greater detail below with reference to FIGS. 10A and10B, blocks for determining and displaying an image on an differentlyshaped/sized image region as described in greater detail below withreference to FIGS. 10A and 10B, and/or blocks for determining anddisplaying visual indicators indicating that the intersection isoff-image and/or off-screen, as described in greater detail below withreference to FIGS. 14A and 14B

FIG. 13 is a flow diagram illustrative of an embodiment of a routine1300 implemented by the system 101 to display at least an intersectionindicator. One skilled in the relevant art will appreciate that theelements outlined for routine 1300 can be implemented by one or morecomputing devices/components that are associated with the system 101,such as the position sensing unit 106, the image guidance unit 104,surgical system 108, and/or the imager 110. Accordingly, routine 1300has been logically associated as being generally performed by the system101. However, the following illustrative embodiment should not beconstrued as limiting. Furthermore, it will be understood that thevarious blocks described herein with reference to FIG. 13 can beimplemented in a variety of orders. For example, the system mayimplement some blocks concurrently or change the order, as desired.

At block 1302, the system determines an emplacement of a first virtualmedical device associated with a first medical device based at least inpart on first emplacement data. As described above, the firstemplacement data can correspond to a device tracker associated with afirst medical device that corresponds to the first virtual medicaldevice, and the first emplacement data can be received from the firstdevice tracker and/or a position sensing unit.

At block 1304, the system determines an emplacement of an image regionbased at least in part on second emplacement data associated with asecond medical device. As described above, the image region can bedetermined directly from the second emplacement data and/or usingemplacement of a second medical device (or virtual medical device) thatis associated with the image region.

At block 1306, the system determines an intersection based at least inpart on the first emplacement data and the second emplacement data. Asdescribed om greater detail above, the intersection can correspond to anintersection between a first emplacement data axis and a secondemplacement data plane, a variance volume/trajectory/axis and an imageplane/region (or portions thereof), etc. In some embodiments, theintersection can correspond to an intersection between the firstemplacement data axis, variance volume, trajectory, or axis with adisplay object, such as the virtual medical device, pre-operative data(e.g., blood vessel, organ, etc.), etc.

At block 1308, the system determines an intersection indicator based atleast in part on the determined intersection. In some embodiments, theintersection indicator is further determined based at least in part onat least one of a variance parameter associated with a device trackerthat is associated with the first medical device and/or anangle-of-approach. In certain embodiments, the size of the intersectionvaries in proportion to the angle-of-approach, as discussed in greaterdetail above.

At block 1310, the system causes one or more displays to display theintersection indicator. As described in greater detail above, theintersection indicator can be displayed in a perspective view in avirtual 3D space. The perspective view can be based at least in part ona point-of-view location.

It will be understood that fewer, more, or different blocks can be usedas part of the routine 1300. For example, in some embodiments, thesystem 101 can cause the one or more displays to display one or morevirtual medical devices, the image region, a medical image, one or moreguidance cues, such as a trajectory indicator, etc. In certainembodiments, the routine 1300 can omit block 1308 and display anintersection indicator (block 1310) based on the determinedintersection. In some embodiments the trajectory indicator can be basedat least in part on the dimensions of the medical device. For example,the trajectory indicator can extend to a location that the medicaldevice can reach based on its length. Furthermore, as non-limitingexamples, the routine 1300 can include any one or any combination ofblocks 1312, 1314, 1316, and/or 1318.

At block 1312, the system can determine and cause one or more displaysto display a variance volume indicator based at least in part on avariance parameter associated with a device tracker that is associatedwith the first medical device.

At block 1314, the system can determine and cause one or more displaysto display a visual indicator that the axis/variance volume does notintersect with the image region based at least in part on adetermination that an axis and/or variance volume associated with thefirst virtual medical device intersects with a location outside theregion but within a viewing area, and.

At block 1316, the system can determine and cause one or more displaysto display a shaded region on the image region based at least in part ona determination that an axis and/or variance volume associated with thefirst virtual medical device intersects with a display object betweenthe virtual medical device and an obstructed portion of the imageregion.

At block 1318, the system can determine and cause one or more displaysto display portions of the image region outside a medical imagedifferently than portions of the image region that overlap with themedical image based at least in part on a determination that the sizeand/or shape of the image region does not correspond to the size and/orshape of the medical image

Terminology

Those having skill in the art will further appreciate that the variousillustrative logical blocks, modules, circuits, and process stepsdescribed in connection with the implementations disclosed herein can beimplemented as electronic hardware, computer software, or combinationsof both. To clearly illustrate this interchangeability of hardware andsoftware, various illustrative components, blocks, modules, circuits,and steps have been described above generally in terms of theirfunctionality. Whether such functionality is implemented as hardware orsoftware depends upon the particular application and design constraintsimposed on the overall system. Skilled artisans can implement thedescribed functionality in varying ways for each particular application,but such implementation decisions should not be interpreted as causing adeparture from the scope of the present invention. One skilled in theart will recognize that a portion, or a part, can comprise somethingless than, or equal to, a whole. For example, a portion of a collectionof pixels can refer to a sub-collection of those pixels.

The various illustrative logical blocks, modules, and circuits describedin connection with the implementations disclosed herein can beimplemented or performed with a processor, a digital signal processor(DSP), an application specific integrated circuit (ASIC), a fieldprogrammable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A processor can be a microprocessor, but in the alternative, theprocessor can be any conventional processor, controller, ormicrocontroller. A processor can also be implemented as a combination ofcomputing devices, e.g., a combination of a DSP and a microprocessor, aplurality of microprocessors, one or more microprocessors in conjunctionwith a DSP core, or any other such configuration.

The steps of a method or process described in connection with theimplementations disclosed herein can be embodied directly in hardware,in a software module executed by a processor, or in a combination of thetwo. A software module can include program instructions that instruct ahardware processor, and can be stored in RAM memory, flash memory, ROMmemory, EPROM memory, EEPROM memory, registers, hard disk, a removabledisk, a CD-ROM, or any other form of non-transitory computer-readablemedium known in the art, as computer-executable instructions. Anexemplary computer-readable storage medium is coupled to the processorsuch the processor can read information and/or computer-executableinstructions from, and write information to, the computer-readablestorage medium. In the alternative, the storage medium can be integralto the processor. The processor and the storage medium can reside in anASIC or FPGA. The ASIC or FPGA can reside in a user terminal, camera, orother device. In the alternative, the processor and the storage mediumcan reside as discrete components in a user terminal, camera, or otherdevice.

Headings are included herein for reference and to aid in locatingvarious sections. These headings are not intended to limit the scope ofthe concepts described with respect thereto. Such concepts can haveapplicability throughout the entire specification.

Conditional language such as, among others, “can,” “could,” “might” or“may,” unless specifically stated otherwise, are otherwise understoodwithin the context as used in general to convey that certain embodimentsinclude, while other embodiments do not include, certain features,elements and/or steps. Thus, such conditional language is not generallyintended to imply that features, elements and/or steps are in any wayrequired for one or more embodiments or that one or more embodimentsnecessarily include logic for deciding, with or without user input orprompting, whether these features, elements and/or steps are included orare to be performed in any particular embodiment.

Language such as the phrase “at least one of X, Y and Z,” and “at leastone of X, Y or Z,” unless specifically stated otherwise, is understoodwith the context as used in general to convey that an item, term, etc.may be either X, Y or Z, or any combination thereof. Thus, such languageis not generally intended to imply that certain embodiments require atleast one of X, at least one of Y and at least one of Z to each bepresent or exclusively X or exclusively Y or exclusively Z.

Unless otherwise explicitly stated, articles such as ‘a’ or ‘an’ shouldgenerally be interpreted to include one or more described items.Accordingly, phrases such as “a device configured to” are intended toinclude one or more recited devices. Such one or more recited devicescan also be collectively configured to carry out the stated recitations.For example, “a processor configured to carry out recitations A, B andC” can include a first processor configured to carry out recitation Aworking in conjunction with a second processor configured to carry outrecitations B and C.

The previous description of the disclosed implementations is provided toenable any person skilled in the art to make or use the presentinvention. Various modifications to these implementations will bereadily apparent to those skilled in the art, and the generic principlesdefined herein can be applied to other implementations without departingfrom the spirit or scope of the invention. Furthermore, althoughdescribed above with reference to medical devices and procedures, itwill be understood that the embodiments described herein can be appliedto other systems in which objects are tracked and virtualrepresentations are displayed on a display and/or systems in whichmultiple objects are displayed on a display within a virtual space, suchas within a virtual 3D space. Thus, the present invention is notintended to be limited to the implementations shown herein but is to beaccorded the widest scope consistent with the principles and novelfeatures disclosed herein.

1. A method, comprising: receiving first emplacement data associatedwith a first device tracker associated with a first medical device;receiving second emplacement data associated with a second devicetracker associated with a second medical device; determining emplacementof a virtual 3D rendering of the first medical device with respect to apoint-of-view location based at least in part on the first emplacementdata; determining emplacement of an image region with respect to thepoint-of-view location based at least in part on the second emplacementdata; determining an intersection of an axis associated with the virtual3D rendering of the first medical device and at least a portion of theimage region based at least in part on the determined emplacement of thevirtual 3D rendering of the first medical device and the determinedemplacement of the image region; determining an intersection indicatorbased at least in part on the determined intersection, a varianceparameter of the first device tracker, and an orientation angle of thevirtual 3D rendering of the first medical device with respect to theimage region; and causing one or more displays to concurrently display:a perspective view of at least a portion of the virtual 3D rendering ofthe medical device in a virtual 3D space based at least in part on thedetermined emplacement of the virtual 3D rendering of the first medicaldevice, a perspective view of at least a portion of the image regionbased at least in part on the determined emplacement of the imageregion, and the intersection indicator, wherein an area of theintersection indicator increases in proportion to a decrease in theorientation angle.
 2. A method, comprising: receiving first emplacementdata associated with a first medical device; receiving secondemplacement data associated with a second medical device; determiningemplacement of a first virtual medical device with respect to apoint-of-view location based at least in part on the first emplacementdata; determining emplacement of an image region with respect to apoint-of-view location based at least in part on the second emplacementdata; determining an intersection based at least in part on the firstemplacement data and the second emplacement data; and causing one ormore displays to concurrently display: a perspective view of at least aportion of the virtual medical device in a virtual 3D space based atleast in part on the determined position and orientation of the virtualmedical device, a perspective view of at least a portion of the imageregion, and an intersection indicator based at least in part on thedetermined intersection and at least one of: an orientation angle of thefirst virtual medical device with respect to the image region or avariance parameter of a device tracker associated with the first medicaldevice.
 3. The method of claim 2, wherein the intersection indicator isbased at least in part on the determined intersection and theorientation angle, and wherein a size of the intersection indicatorincreases in proportion to an increased difference between thedetermined orientation angle and a threshold angle.
 4. The method ofclaim 2, wherein determining the intersection comprises determining anintersection between an axis associated with the first medical deviceand a plane associated with the second medical device.
 5. (canceled) 6.The method of claim 2, wherein at least a portion of the image region isdisplayed differently based at least in part on a determination that theintersection is outside the image region.
 7. (canceled)
 8. The method ofclaim 2, wherein causing the display to display further comprisescausing the one or more displays to display an arrow indicating adirection of a virtual medical device and/or the intersection indicatorbased at least in part on a determination that the intersection isoutside the image regi.
 9. The method of claim 2, wherein determiningthe intersection comprises determining an intersection between an axisassociated with the first medical device and at least a plurality ofplanes of the image region.
 10. (canceled)
 11. (canceled)
 12. The methodof claim 2, wherein determining the intersection comprises determiningan intersection between an axis associated with the first medical deviceand a second virtual medical device associated with the second medicaldevice.
 13. The method of claim 12, wherein at least a portion of theintersection indicator is displayed on the second virtual medicaldevice.
 14. The method of claim 2, wherein the intersection is a firstintersection, the method further comprising determining a secondintersection based at least in part on the first emplacement data andthird emplacement data associated with a display object, wherein theintersection indicator is displayed based at least in part on the firstintersection and the second intersection.
 15. The method of claim 14,further comprising determining an obstructed portion of the intersectionindicator based at least in part on the second intersection, wherein theobstructed portion of the intersection indicator is not displayed or isdisplayed differently from an unobstructed portion of the intersectionindicator.
 16. The method of claim 14, further comprising determining anobstructed portion of the image region based at least in part on thesecond intersection, wherein the obstructed portion of the image regionis displayed differently from an unobstructed portion of the imageregion.
 17. The method of claim 2, wherein at least a portion of theintersection indicator is parallel to a side of the image region. 18.The method of claim 2, wherein a first portion of the intersectionindicator is parallel to a first side of the image region and a secondportion of the intersection indicator is parallel to a second side ofthe image region.
 19. The method of claim 2, wherein the intersectionindicator comprises two parallel lines and rounded ends.
 20. The methodof claim 2, wherein the image region is shaped differently from amedical image mapped to the image region, and wherein portions of theimage region that do not include the medical image are displayeddifferently than portions of the image region that include the medicalimage.
 21. The method of claim 2, further comprising: determining avariance volume based at least in part on the variance parameter of thedevice tracker and the emplacement of the virtual 3D rendering of thefirst medical device, wherein determining the intersection is based atleast in part on an intersection of the variance volume and the imageregion, and wherein causing the display to display the intersectionindicator is based at least in part on the intersection of the variancevolume and the image region.
 22. A system, comprising one or moreprocessors communicatively coupled with one or more displays; and anon-transitory computer-readable storage medium storingcomputer-executable instructions that when executed by the one or moreprocessors cause the one or more processors to: receiving firstemplacement data associated with a first medical device; receivingsecond emplacement data associated with a second medical device;determining emplacement of a first virtual medical device with respectto a point-of-view location based at least in part on the firstemplacement data; determining emplacement of an image region withrespect to a point-of-view location based at least in part on the secondemplacement data; determining an intersection based at least in part onthe first emplacement data and the second emplacement data; and causingone or more displays to concurrently display: a perspective view of atleast a portion of the virtual medical device in a virtual 3D spacebased at least in part on the determined position and orientation of thevirtual medical device, a perspective view of at least a portion of theimage region, and an intersection indicator based at least in part onthe determined intersection and at least one of: an orientation angle ofthe first virtual medical device with respect to the image region or avariance parameter of a device tracker associated with the first medicaldevice.
 23. The system of claim 22, wherein the intersection indicatoris based at least in part on the determined intersection and theorientation angle, and wherein a size of the intersection indicatorincreases in proportion to an increased difference between thedetermined orientation angle and a threshold angle.
 24. The system ofclaim 22, wherein the intersection is a first intersection, and whereinthe computer-executable instructions further cause the one or moreprocessors to determine a second intersection based at least in part onthe first emplacement data and third emplacement data associated with adisplay object, wherein the intersection indicator is displayed based atleast in part on the first intersection and the second intersection.25.-44. (canceled)